Flow control is the management of data flow between nodes in a network so that the data can be handled at an efficient pace. As the traffic load increases in a network, resource contention increases. When too much data arrives before a device can handle it, data overflow can result, meaning the data is either lost or must be retransmitted. Under congested network conditions, flow control can help to smooth network operation, improve efficiency and effective bandwidth, and reduce average packet latency.
A variety of flow control mechanisms are known in the art, falling generally into the categories of dynamic control and static rate control. Dynamic flow control methods track network congestion in real time, and regulate the flow of data based on current congestion levels. Static rate control, on the other hand, uses pre-configured values, which are typically set by a network management entity based on known capacities of network nodes and links. Static rate control typically requires that network end-nodes keep the rates at which they transmit data into the network below certain predetermined maxima.
For example, in InfiniBand™ (IB) switch fabrics, static rate control is used to reduce congestion that is caused when a high-speed device injects packets onto a path within a subnet at a rate that exceeds the ability of the path or destination device to transfer or receive the packets. The operation of IB fabrics is described in detail in the InfiniBand Architecture Specification, Release 1.0 (October, 2000), which is incorporated herein by reference. This document is available from the InfiniBand Trade Association at www.infinibandta.org. Computing devices (host processors and peripherals) connect to the IB fabric via network interface adapters, which are referred to in IB parlance as channel adapters. As in other packet networks, each IB packet transmitted by a channel adapter carries a media access control (MAC) destination address, referred to as a Destination Local Identifier (also known as a Destination LID, or DLID). The LID is used by nodes in the fabric to relay the packet to its destination.
Channel adapters in IB fabrics are required to implement static rate flow control, as described in section 9.11 of the IB specification. For example, when a channel adapter with a 10 Gbps output link transmits packets to another channel adapter with a 2.5 Gbps input link through an intermediate switch, the transmitting channel adapter must not send out the packets at a rate any faster than 2.5 Gbps, in order to prevent the slower link from being overrun. Therefore, static rate limitations are imposed on channel adapter ports that are configured for injection of packets into the fabric at a rate greater than 2.5 Gbps (the minimum basic rate in IB fabrics). Such a port must not schedule a packet for injection into its local subnet until an appropriate amount of time has passed since the last packet was scheduled for injection from this source port to the same destination port.
The time that the input port must wait between packets is programmable, depending on transmission and reception rate characteristics. According to the IB specification, the time to wait between scheduling successive packets destined for the same DLID and originating from the same port is determined by the Inter Packet Delay (IPD). Specifically, if a packet b is to be sent to the same DLID and using the same source port as packet a, then packet b must not be scheduled until a tine Ts has passed since packet a was scheduled. Ts is calculated as (IPD+1) multiplied by the time it takes to transmit packet a. The time it takes to transmit the packet depends on the length of the packet and the port speed. The IPD value is an 8-bit integer, which is interpreted as depicted in Table I below (which is adapted from Table 63 in the IB specification):
TABLE IINTER PACKET DELAYIPDRateComment0100%Suited for matched links1 50%2 33%Suited for 30 Gbps to 10 Gbps conversion3 25%Suited for 10 Gbps to 2.5 Gbps conversion11 8%Suited for a 30 Gbps to 2.5 Gbps conversion
Although the IB specification defines the required delay to be introduced between packets for the purpose of static rate flow control, it does not suggest how such flow control should be implemented. Typically, a channel adapter is required to handle multiple transport service instances (known as queue pairs) simultaneously, each generating packets to be sent over the fabric to different DLIDs. For some of these DLIDs, flow control may be required, while for others it may not. When flow control is required, different values of the IPD may apply to different DLIDs. Even for a single DLID, the actual delay time between packets varies depending on the lengths of the packets transmitted. Under these conditions, scheduling the packets for transmission can place a heavy computational burden on the output circuits of the channel adapter, particularly when the high speed of the output port (10–30 Gbps) is considered.